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(57) Abstract: A method of processing image data 
lor display on a pixelated imaging device is dis- 
closed. The method comprises: pre -compensation 
filtering an image input to produce pre-compensa- 
tion filtered pixel values, the pre- compensation fil- 
ter having a transfer function that approximates the 
function that equals one divided by a pixel trans- 
fer function; and displaying the pre -compensation 
filtered pixel values on the pixelated imaging de- 
vice. In another disclosed method, the method fur- 
ther comprises: pre-compensation filtering an im- 
age input for each of a plurality of superposed pix- 
elated imaging devices, at least two of which are 
unaligned, to produce multiple sets of pre-compen- 
sation filtered pixel values; and displaying the mul- 
tiple pre-compensation filtered pixel values on the 
plurality of superposed pixelated imaging devices. 
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SYSTEM AND METHOD FOR OPTIMIZING IMAGE RESOLUTION USING 

PIXELATED IMAGING DEVICES 



5 CROSS-REFERENCE TO RELATED APPLICATION 

This application claims the benefit of our provisional application serial 
number 60/179,762, filed February 2, 2000, the disclosure of which is hereby 
incorporated herein by reference. 

10 

FIELD OF THE INVENTION 

The invention generally relates to systems and methods for optimizing the 
resolution of graphical displays, and more particularly the invention relates to 
15 systems and methods for optimizing the resolution of pixelated displays. 

BACKGROUND OF THE INVENTION 

Graphical display engineers continue to minimize pixel hardware size. 
20 However, for any given minimum pixel size, there is an ongoing need to optimize 
display resolution. 

SUMMARY OF THE INVENTION 



25 In one embodiment of the invention, a method of processing image data for 

display on a pixelated imaging device comprises: pre-compensation filtering an 
image input to produce pre-compensation filtered pixel values, the pre- 
compensation filter having a transfer function that approximates the function that 
equals one divided by a pixel transfer function; and displaying the pre- 

30 compensation filtered pixel values on the pixelated imaging device. 



In another embodiment of the invention, a method further comprises: 

1 
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pre-compensatiori filtering an image input for each of a plurality of superposed 
pixelated imaging devices, at least two of which are unaligned, to produce 
multiple sets of pre-compensation filtered pixel values; and displaying the 
multiple pre-compensation filtered pixel values on the plurality of superposed 
5 pixelated imaging devices. 

In a further embodiment of the invention, a method further comprises: 
displaying the multiple pre-compensation filtered pixel values on six imagers, the 
six imagers being positioned into four phase families, the first and third phase 
10 families corresponding to separate green imagers, the second and fourth phase 
families corresponding to separate sets of aligned blue and red imagers. 

Further related system and method embodiments are also disclosed. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing description of various embodiments of the invention should 
be appreciated more fully from the following further description thereof with 
reference to the accompanying drawings wherein: 

20 

Fig. 1 shows an imager arrangement for optimizing display resolution in 
accordance with an embodiment of the invention; 

Figs. 2A and 2B show schematic block diagrams of methods of processing 
25 image signals to optimize image resolution, in accordance with two different 
embodiments of the invention; 

Figs. 3A and 3B detail implementation of pre-compensation filters in 
accordance with two different embodiments of the invention; 

30 
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Fig. 4 shows a one-dimensional pixel transfer function; 

Fig. 5 shows a set of transfer functions, determined in accordance with an 
embodiment of the invention; 

5 

Fig. 6 shows a two-dimensional pixel transfer function; 

Fig. 7 shows a two-dimensional pre-compensated pixel transfer function, 
determined in accordance with an embodiment of the invention; 

10 

Fig. 8 shows an extended pre-compensation filter tranfer function, in 
accordance with an embodiment of the invention; and 

Fig. 9 shows a set of multiple unaligned imagers for optimizing image 
1 5 appearance to the human eye, in accordance with an embodiment of the 
invention. 

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 

20 Fig. 1 shows an imager arrangement for optimizing display resolution in 

accordance with an embodiment of the invention. Each of pixelated imaging 
devices 101-104 includes pixel hardware that forms an array of regular polygons, 
tiling a planar area. To optimize display resolution, multiple pixelated imaging 
devices 101-104 are superposed in an unaligned fashion, to form a combined 

25 display device 109. Each successive imaging device of the four superposed 

imaging devices 101-104 is offset by one-quarter of the pixel dimension S, in both 
the vertical and horizontal directions. 

Individual pixel-like features 111 of the resulting combined display device 
30 109 have a minimum dimension, S/ 4, that is one-quarter the minimum 
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dimension, S, of the actual pixels of each separate imaging device 101-104. The 
pixel-like features 111 of the combined display device 109 thus have a square area 
that is one-sixteenth that of the actual pixels of each separate imaging device 101- 
104. The size reduction may be seen in Fig. 1 by comparing the size of individual 
5 pixel-like feature 110 of the combined display device 109, with the size of 
individual pixels 105-108 of the separate pixelated imaging devices 101-104. 

The unaligned superposition of Fig. 1 thus allows increased display 
resolution for a given minimum pixel dimension S, which may, for example, be 
10 the smallest pixel dimension that is presently capable of being implemented in 
hardware for a single separate display. 

In Fig. 1, the polygons of imaging devices 101-104 are square, but they may 
also be rectangular, or any other shape, in accordance with an embodiment of the 

15 invention. While four pixelated imaging devices are shown in Fig. 1, any number 
may be used. In addition, the lack of alignment of the separate imaging devices 
may be produced by a variety of different methods, in accordance with other 
embodiments of the invention. For example, pixelated imaging devices with 
square or rectangular pixels may be spatially shifted by a different amount in the 

20 horizontal direction than in the vertical direction. Two or more imaging devices 
may be aligned with each other, with no spatial shift, while others are unaligned 
with each other, in the same display. A lack of alignment may also be produced 
by an optical or physical change between the separate imaging devices; or by 
changing the separate imaging devices' scale, rotational angle, aspect ratio, or 

25 liner offset with respect to each other. In another embodiment, a time- 
multiplexed imager may be used to produce the same effect as is produced by 
superposing multiple separate imaging devices: the imager or optics of the time- 
multiplexing imager moves to a different position when transmitting a signal that 
corresponds to each of the separate imaging devices. 

30 
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The below chart shows a comparison of three displays: in the "Aligned" 
array, three pixelated imagers are fully aligned, with no offset as shown in Fig. 1. 
In the "1/2 offset" array, three imagers are used, one having green color 
information, and the other two having blue and red information; the blue and red 
5 arrays are aligned with each other, but both diagonally offset by Vi of the pixel 

diagonal spacing from the green imager. In the "1/4 offset" array, six imagers are 
used, two for red information, two for green, and two for blue, as will be further 
described below in connection with Fig. 9. The comparison shows the increase in 
effective resolution, and decrease in imager visibility, that results from un- 
10 aligning superposed imagers, as in the embodiment of Fig. 1: 

Aligned Vi offset V4 offset 



- Array Size 1280x1024x3 1280x1024x3 1280x1024x6 

15 

Hor. Addressable 1280 1280x2 1280x4 

Vert. Addressable 1024 1024x2 1024x4 

20 Effective Resolution 1280x1024 1800x1400 2560x2048 

Imager visibility 1 Vi V4 

In the above table, "Imager Visibility" is used to refer to the relative 
25 visibility of the imager as compared with the image when viewing the image of a 
close distance. As can be seen, unaligned imagers reduce the imager visibility, 
which is caused by the imagers 7 finite resolution and interpixel gaps; in general 
the reduction of imager visibility is proportional to the number of offsets used. 

30 Figs. 2A and 2B show schematic block diagrams of methods of processing 

5 
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image signals to optimize image resolution, in accordance with two different 
embodiments of the invention. In a pixelated grid display, frequency response 
tapers off to zero at the Nyquist frequency of the 2D box filter implemented by the 
filter. This response varies according to radial frequency direction, and phase 
5 relationship to the pixel grid. To improve the image quality achieved by such a 
pixelated display, an embodiment according to the invention oversamples an 
image relative to the display, and generates pixel values by using a two- or three- 
dimensional pre-compensation filter. The filter combines radial bandlimiting, to 
avoid aliasing, with pre-compensation for the imperfect and directional frequency 
10 response function of the display. 

In step 221 of Fig. 2A, an image input is fed to a pre-compensation filter. 
The image input may be in any of a variety of formats, including, for example, 
HDTV, NTSC, PAL etc. In step 222 of Fig. 2A, the pre-compensation filter 
15 transforms the image input and feeds the resulting output directly to a pixelated 
imaging device, where an image is displayed in step 223. In the embodiment of 
Fig. 2A, the pixelated imaging device may be a conventional display, so that the 
pre-compensation filter improves the sharpness and resolution of a conventional 
display in a fashion described further below. 

20 

By contrast, in step 225 of the embodiment of Fig. 2B, the pre-compensation 
filter transforms the image input into a set of pre-compensated image signals, and 
feeds each pre-compensated signal to a different imaging device of a combined set 
of superposed, unaligned pixelated imaging devices. For example, the pre- 
25 compensation filter may feed a separate pre-compensated output signal to each of 
the imaging devices 101-104 that form the combined pixelated imaging device 109 
of the embodiment of Fig. 1. In step 226 of Fig. 2B, a resulting image is displayed 
on the combined set of superposed, unaligned pixelated imaging devices. 

30 Figs. 3A and 3B further detail implementation of pre-compensation filters 
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in accordance with two different embodiments of the invention. The filters may 
be used, for example, as pre-compensation filters in the embodiments of Figs. 2A 
and 2B, respectively. 

5 In step 331 of Fig. 3A, the transfer function of an individual pixel is 

determined. This may be performed by determining the Fourier Transform (or 
other frequency-domain representation) of the pixel's impulse response. 

For example, a pixel could be modeled in one dimension as having a 
10 "boxcar" impulse response, equal to 1 at the pixel's spatial location and 0 
elsewhere. A transfer function for such a pixel is given by: 



H[x] = Sinc[x] 



l,x= 0 
sin[jc] 



,jc * 0 

{Eq. 1} 



for spatial frequency x. This transfer function is shown in the graph of Fig. 
15 4, with frequency x on the x-axis in radians, and the pixel's transfer function on 
the y-axis. 

Analogously, a pixel could be modeled in two dimensions as a square finite 
impulse response filter with unity coefficients inside the pixel's spatial location 
20 and zero coefficients elsewhere. A transfer function for such a pixel is given by: 

H[u,V]= Sinc[u]*Sinc[V] 

{Eq. 2} 

with Sine as defined for Equation 1, and "*" denoting convolution. 

25 Next, in step 332 of Fig. 3A, the pixel's transfer function is used to 

determine a transfer function for the pre-compensation filter. In one embodiment 
according to the invention, the pre-compensation filter is chosen such that its 

7 
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transfer function satisfies the expression: 

{Pre-compensation filter transfer function} = 

{ 1 / pixel transfer function } 
{Eq. 3} 



although other relations that give similar results will be apparent to those 
of ordinary skill in the art. 

Next, in step 333 of Fig. 3 A, an adjusted transfer function for the pre- 
compensation filter is determined. This step may involve, for example, gain- 
limiting the pre-compensation filter's transfer function; or clipping off its values at 
aliasing frequencies. For example, using the one-dimensional example above, an 
example of a gain-limited and clipped pre-compensation filter transfer function is 
given by: 



with G being a gain factor that could be set, for example, to equal 4. 

Using the analogous two-dimensional example given above, an example of 
a two-dimensional gain-limited and clipped pre-compensation filter transfer 
function is given by: 



Hg[x] = 



0, Abs[x] > 7t\ 

Sign[Sinc[x)](G - ((G / 2) A 2) Abs[Sinc[x]]) y Abs[Sinc[x]] < 2 1 G; 



1 / Sinc[x], otherwise 



{Eq. 4} 



Hc[uy] = 



0, Abs[u\ > n\ 
0,Abs[V]> n\ 

Sign[Sinc[u] * Sinc[v]](G - ((G / 2) A 2) Abs[Sinc[u] * Sinc[V}\ 



for_ Abs[Sinc[u] * Sinc[V]] < 2 / G; 



1 / (Sinc[u] * Sinc[V]), otherwise 



{Eq. 5} 



with G being a gain factor that could be set, for example, to equal 4. 
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Next, in step 334 of Fig. 3A, the adjusted transfer function of the pre- 
compensation filter calculated in step 333 is used to calculate individual 
coefficients of a pre-compensation finite impulse response filter. This is 
performed by a transform back into the spatial domain (out of the frequency 
domain), using, for example, an inverse Fourier transform. In the two- 
dimensional example given above, an individual pre-compensation filter 
coefficient can be calculated by the expression: 



Next, in step 335 of Fig. 3A, the entire pre-compensation finite impulse 
response filter is determined, by combining individual coefficients as calculated in 
step 334 into a single array formed from coefficients that correspond to each pixel 
location in the display. Thus, for example, Equation 6 could be used to calculate a 
coefficient for each pair (m,n) of a coordinate system covering a two-dimensional 
pixelated imaging device. 

Next, in step 336 of Fig. 3A, the pre-compensation finite impulse response 
filter determined in step 335 is used to transform image input data; and, in step 
337, the resulting filtered image is displayed. 

In this fashion a pre-compensation filter of the embodiment of Fig. 3A may 
be used to improve the resolution of a pixelated imaging device, which may be, 
for example, a conventional pixelated display. The improvement in image 
appearance is evidenced. by Fig. 5, which shows a graph of transfer functions in 
accordance with the one-dimensional example used above. The pixel transfer 
function H[x] 550 is plotted on the same axes as the transfer function 552 of the 
pre-compensation finite impulse response filter formed from coefficients C[m,n]. 
Transfer function 551 is the pre-compensated pixel transfer function that results 




{Eq. 6} 
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from transforming an image input with the pre-compensation filter before feeding 
it to the pixelated display. The more // square-shouldered ,/ transfer function 551 of 
the pre-compensated display, as compared with the rounded transfer function 550 
of the display without pre-compensation, is evidence of the improved resolution 
5 brought about by the embodiment of Fig. 3A. A similar contrast may be observed 
by comparing the shape of the two-dimensional pixel transfer function of a 
display that lacks pre-compensation (Fig. 6) with the "square-shouldered" pixel 
transfer function of a pre-compensated display (Fig. 7). 

10 In an alternative version of the embodiment of Fig. 3 A, the pre- 

compensation filter need not be "clipped off" within the frequency band shown 
above; instead, it may have an extended frequency range. Fig.8 shows a pixel 
transfer function 801, an extended pre-compensation filter transfer function 802, 
and the "square-shouldered" transfer function 803 that results from using the pre- 

15 compensation filter 802 to filter image input. 

Whereas the embodiment of Fig. 3A may be used with a single pixelated 
imaging device, the embodiment of Fig. 3B may be used with multiple, 
superposed imaging devices, such as the unaligned imaging devices of the 
20 embodiment of Fig. 1. 

Steps 338-340 of Fig. 3B mirror steps 331-333 of Fig. 3A. In step 341 of Fig. 
3B, however, individual coefficients of pre-compensation filters are calculated in a 
similar fashion to that of step 334 of Fig. 3A, but by also taking into consideration 

25 the spatial phase shift of the unaligned imagers to which the filters correspond. 
For example, a set of four pre-compensation filters would be used to filter inputs 
to the four unaligned imagers 101-104 of the embodiment of Fig. 1, with one pre- 
compensation filter corresponding to each one of the imagers 101-104. When 
calculating coefficients for the pre-compensation filter for imager 101, a phase- 

30 shift of zero would be included; but when calculating coefficients for the filter for 

10 
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imager 102, a diagonal phase-shift of one-quarter of a pixel would be included in 
the calculations; and so on. In the expression of Equation 6, for example, having 
the values of m and n both range from -3 % to +4 X A could be used to calculate 
coefficients of a filter corresponding to a one-quarter diagonal pixel offset imager; 
5 as compared with ranges from -4 to +4 for an imager with zero diagonal offset, 
and -3 Vi to +4 Vi for an imager with a one-half diagonal pixel offset. 

In step 342, the individual coefficients calculated in step 341 are used to 
calculate an entire pre-compensation finite impulse response filter , for each 
10 spatially phase-shifted pixelated imaging device. Arrow 343 indicates that 

individual coefficients are calculated, in step 341, until the coefficients for all pre- 
compensation filters are filled. For example, four filter arrays would be filled 
with coefficients, to create four pre-compensation filters for the unaligned imagers 
of the embodiment of Fig. 1. 

15 

In step 344, each pre-compensation filter is used to transform image input 
data for its corresponding phase-shifted pixelated imaging device. In step 345, a 
superposed, pre-compensation filtered image is displayed. 

20 Fig. 9 shows a method in accordance with an embodiment of the invention 

that optimizes image resolution by adapting the previously described methods to 
the human eye's optics. The eye's perception of Red and Green is 1/3 its 
perception of luminance, and its perception of Blue and Yellow is 1/8 its 
perception of luminance. Thus, high-frequency information in the luminance 

25 component of an image is more valuable than information in the chrominance 
components, for optimizing an image's appearance. 

The embodiment of Fig. 9 uses a set of six superposed, unaligned imagers 
to take into account these aspects of the eye's perception. Two imagers are fed 
30 red chrominance information, two are fed green chrominance information, and 

11 
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two are fed blue chrominance information. Because of the eye's different 
weighting of different colors, the six imagers are combined into four phase 
families: a first, single green imager 901 has zero phase offset; a second imager 902 
comprising a blue imager aligned with a red imager has a l A diagonal pixel offset 
5 as compared with the single green imager 901; a third imager 903 comprising a 
single green imager has a Vi diagonal pixel offset as compared with the single 
green imager 901; and a fourth imager 904 comprising a blue imager aligned with 
a red imager has a 3 A diagonal pixel offset as compared with the single green 
imager 901. 

10 

The embodiment of Fig. 9 may be operated in a similar fashion to that 
described in Figs. 2B and 3B, or may be fed phase-shifted signals without pre- 
compensation. 

15 In another embodiment according to the invention, a perception-based 

representation of the image - such as a YUV or HIS representation, for example, 
instead of an RGB representation - is processed by its own reconstruction filter. 
The output of the filter yields the appropriate perception-based pixel value for 
each element of each grid; this is then converted to the appropriate color value for 

20 each element of each grid. 

While the embodiments described above have been discussed in terms of 
image projection on pixelated displays, similar methods may be used for image 
sensing and recording, in accordance with an embodiment of the invention. 

25 

Multiple unaligned sensors may be set up, in an analogous fashion to the 
multiple displays of Fig. 1; or one image may be split among multiple real or time- 
multiplexed imagers by beam splitters. 



30 



For color applications, each imager may operate in one color frequency 
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band. For example, a set of six unaligned color sensors may be implemented in a 
similar fashion to that described for Fig. 9. As with the embodiments of Figs. 2B 
and 3B, the image inputs from each sensor device may be pre-compensation 
filtered. 

5 

In addition, however, the separate viewpoint provided by each sensor may 
be considered as a single 2D-filtered view of an infinite number of possible image 
signals, that provides constraints on the image to be displayed. A displayed 
image is then calculated by determining the lowest energy signal that satisfies the 
10 constraints established by the signals from the various separate sensors. That is, 
the energy 

X X Abs[S[m,n]y2 

m n 

{Eq. 7} 

is minimized for proposed signals S[m,n] that satisfy the boundary 
conditions established by sensor image signals S 1 [m / n]- ...S k [m,n], for k imagers. 
15 The proposed signal S[m,n] that provides the minimum energy value is then used 
as the sensed signal for display. 

In another embodiment, a color camera is implemented by dividing the 
visible spectrum for each physical sensor using a diachroic prism. In one 

20 example, six imagers are used, with a prism dividing the image into six color 

frequency ranges. Information from each color frequency range is then supplied 
to a displaced imager. The lowest energy luminance and color difference signals 
are then solved. These signals satisfy the constraints generated by the six imager 
signals and their known 2D frequency response and phasing. In addition to the 

25 frequency response and phasing of each imager, the sagital and tangential 
frequency response of the optics at that light frequency may be included in 
calculations, to correct for the Modular Transfer Function (MTF) of the optics. 
The contribution of each of the six color bands is then weighted for human 
perception of luminance and of the Cb and Cr signals. 

13 
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In another embodiment, a playback device is implemented. The playback 
device filters and interpolates the original signal to provide the correct transfer 
function and signal value at the location of each pixel on each imager. If more 
than one imager is used for each color component, the component image energy 
may be divided and weighted for perception among the imagers. If each color 
component is divided into separate color frequencies, the image energy may be 
divided among those components and weighted by perception. 

Another embodiment comprises a recording device. To record the signal, 
there are two approaches. One is to record each imager's information as a 
separate component. This preserves all of the information. The other alternative 
is to record a combined high-frequency luminance signal and two combined color 
difference signals. If three to six imagers are used, good results can be obtained 
by recording a luminance signal with twice the resolution in both dimensions as 
the two color difference signals. 

In a polarized light embodiment, multiple imagers are operated with two 
clases of polarized light. Separate eye views are supplied to imagers, so that a 
single projection device gives a three-dimensional appearance to the projected 
image. 

An embodiment of the invention also provides a technique for 
manufacturing imagers for use with the embodiments described above. In 
accordance with this embodiment, if color component imagers are assembled with 
little concern to their precise orientation, or response, spot response sensors (for 
projection), or calibrated spot generators (in the case of a camera), allow 
inspection at the geometric extremes of the image. This inspection, combined 
with a hyperaccuity-based signal processing approach, determine exact placement 
phase, scale, rotation and tilt of each manufactured display. If tilt is not required, 
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two sensors suffice. 

In one embodiment, such sensors can be used in manufacturing to set 
placement parameters. In another embodiment, such sensors are used in the 
product to automatically optimize response for component grid placement. In 
this embodiment, the sensors can also be used for automatic color correction and 
white balance for the current environment. The process and the feedback 
hardware required can be generalized to compensate for manufacturing tolerance, 
operational, or calibration requirements. In the most general case, automatic 
compensation requires a full image sensor for a projector, or a reference image 
generator for a camera. In this case, flat field, black field, linearity, color shift, 
geometric distortion, and modulated transfer function can all be compensated for. 

Some embodiments of the invention may be implemented, at least in part, 
in any conventional computer programming language comprising computer 
program code. For example, preferred embodiments may be implemented, at 
least in part, in a procedural programming language (e.g., "C") or an object 
oriented programming language (e.g., "C++")- Alternative embodiments of the 
invention may be implemented, at least in part, as preprogrammed hardware 
elements (e.g., application specific integrated circuits, FPGAs, and digital signal 
processors), or other related components. 

The present invention may be embodied in other specific forms without 
departing from the true scope of the invention. The described embodiments are 
to be considered in all respects only as illustrative and not restrictive. 
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I claim: 

1. A method of processing image data for display on a pixelated imaging 
device, the method comprising: 

5 pre-compensation filtering an image input to produce pre-compensation 

filtered pixel values, the pre-compensation filter having a transfer 
function that approximates the function that equals one divided by a 
pixel transfer function; and 
displaying the pre-compensation filtered pixel values on the pixelated 
10 imaging device. 

2. A method according to claim 1, in which the method further comprises: 
pre-compensation filtering an image input for each of a plurality of 

superposed pixelated imaging devices, at least two of which are 
15 unaligned, to produce multiple sets of pre-compensation filtered 

pixel values; and 

displaying the multiple pre-compensation filtered pixel values on the 
plurality of superposed pixelated imaging devices. 

20 3. A method according to claim 2, in which the method further comprises: 
displaying the multiple pre-compensation filtered pixel values on six 

imagers, the six imagers being positioned into four phase families, 
the first and third phase families corresponding to separate green 
imagers, the second and fourth phase families corresponding to 

25 separate sets of aligned blue and red imagers. 
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